We have now completed data collection for the first phase of this project. We extracted author-facing statistical guidance provided by the top-15 journals (by Impact Factor) in each of 22 scientific disciplines (i.e., total N = 330 journals). The preregistered study protocol is available here. Below we present a preliminary analysis of the phase one data and highlight some issues to discuss as a team.
Operational definition of statistical guidance: any advice or instruction related to the appropriate selection, implementation, reporting, or interpretation of statistical analyses.
We recorded whether each journal:
If any statistical guidance was provided by a journal:
Note: When journals referred to guidance in external sources, we have recorded the names of the sources, but we have not currently extracted guidance from those sources. We return to this point in the issues to address section (Section 4) at the end of this document.
All data extraction was performed by a first coder (DS, MM, TB, MSH) and a second coder (TEH) with any coding differences resolved through discussion.
Note: We found that some publishers provided statistical guidance that was shared across their portfolio of journals (specifically, 31 Nature journals, 12 Cell journals, and 2 Frontiers journals). Shared publisher guidance was coded by two team members (as above), but is represented in our data and analysis multiple times - once for each individual journal it applies to. For example, the Cell journals guidance is inherited by 12 journals so appears 12 times in the dataset. Unless stated otherwise, publisher guidance is treated as journal-specific guidance (i.e., not external guidance) in the analyses presented below.
Figure 1 shows the percentage of journals in each scientific discipline (denominator is N = 15) and overall (denominator is N = 330) that (a) offered any statistical guidance; (b) had a dedicated statistical guidance section. For tabular data, see Appendix 5.
You can see that around half (48%) of journals offered any statistical guidance. Note that this includes 32 journals which only referred to statistical guidance in external sources (reporting guidelines or academic papers). Just over quarter of journals had a dedicated statistical guidance section of their author instructions (28%). In two fields (Computer Science and Maths), no journals offered any statistical guidance. Journals in health-related fields were more likely to offer statistical guidance and have dedicated statistical guidance sections. Notably, 100% of the journals in clinical medicine offered some statistical guidance.
Figure 1: Percentage of journals offering statistical guidance by scientific discipline (N = 15; represented by coloured dots) and overall (N = 330; represented by black diamond). Graphs ordered from left to right and top to bottom in order of highest proportion overall across scientific disciplines.
Figure 2 shows the percentage of journals in each scientific discipline (denominator is N = 15) and overall (denominator is N = 330) that mentioned each of the twenty pre-specified statistical topics. For tabular data, see the Appendix (Section 5). Please see Section 4 (issues) at the end of this document for an important note regarding the topic “prespecification of analyses”.
Figure 2: Percentage of journals offering guidance on twenty statistical topics by scientific discipline (N = 15; represented by coloured dots) and overall (N = 330; represented by black diamond). For presentational purposes, disciplines offering no guidance on individual topics are shown, but not labelled. Two scientific disciplines in which no journals offered any statistical guidance at all are not shown. Graphs ordered from left to right and top to bottom in order of highest proportion overall across scientific disciplines.
You can see that some topics (e.g., confidence intervals, p-values) are much more likely to be mentioned than other topics (e.g., handling outliers, categorisation of continuus data). Journals in health-related fields, particularly clinical medicine, are often more likely to mention these topics than journals in other fields.
For the 128 journals that offered some internal guidance (i.e., excluding those that offered no guidance at all and those that only referred to external sources), the histogram in Figure 3 illustrates that the maximum number of topics mentioned by an individual journal was 15. There were 9 journals that did not mention any of our prespecified topics (but provided statistical guidance on other topics). The median topics mentioned was 6.
Figure 3: Histogram showing how many of the twenty statistical topics were mentioned by each journal
In the grid below (Figure 4), you can see which of the twenty topics were mentioned by each journal. Journals that shared publisher guidance are represented by one row (per publisher). Journals that did not offer any statistical guidance at all, or journals that referred only to guidance in external sources, are not shown.
Figure 4: Grid diagram showing whether each journal provdied guidance on each of twenty prespecified statistical topics
| topic | example guidance |
|---|---|
| baseline covariates | Statistical tests, along with reported P values, for comparing groups at baseline are not necessary unless there is a strong reason to include them. (JOURNAL OF ALLERGY AND CLINICAL IMMUNOLOGY) |
| baseline covariates | For randomized trials using parallel-group design, there is no validity in conducting hypothesis tests regarding the distribution of baseline covariates between groups; by definition, these differences are due to chance. Because of this, tables of baseline participant characteristics should not include P values or statements of statistical comparisons among randomized groups. Instead, report clinically meaningful imbalances between groups, along with potential adjustments for those imbalances in multivariable models (JAMA Neurology) |
| Bayesian statistics | Authors are expected to apply the most appropriate statistical tools for data analysis, and it is acceptable to present results from frequentist, information-theory, and Bayesian approaches in the same manuscript. (BRAIN) |
| Bayesian statistics | Inherited Nature [LSR] guidance: For Bayesian analysis, information on the choice of priors and Markov chain Monte Carlo settings (Scientific Data) |
| categorisation of continuous data | Categorizing of continuous data (e.g. into quartiles, quintiles) is discouraged. It leads to a loss of information, usually needs more complicated methods than for continuous data and introduces demarcations which are valid only for this particular study. (EUROPEAN HEART JOURNAL) |
| categorisation of continuous data | If the data are appropriate, age grouping should be mid-decade to mid-decade or in five-year age groups (e.g. 35-44 or 35-39, 40-44, etc, but not 20-29, 30-39 or other groupings. (INTERNATIONAL JOURNAL OF EPIDEMIOLOGY) |
| confidence intervals | Descriptive statistics should include a clearly labelled measure of centre (such as the mean or the median), and a clearly labelled measure of variability (such as standard deviation or range). Ranges are more appropriate than standard deviations or standard errors for small data sets. Standard error or confidence interval is appropriate to compare data to a control. (EMBO JOURNAL) |
| confidence intervals | For intervention studies, the abstract should include the primary outcome expressed as the difference between groups with a confidence interval on that difference (absolute differences are more useful than relative ones). Secondary outcomes can be included as long as they are clearly marked as secondary and all such outcomes are reported. (LANCET ONCOLOGY) |
| effect sizes | Effect size and biological importance must not be confused with statistical significance. (ECOLOGICAL MONOGRAPHS) |
| effect sizes | When possible, the editors prefer that absolute event counts or rates be reported before relative risks or hazard ratios. The goal is to provide the reader with both the actual event frequency and the relative frequency. Odds ratios should be avoided, as they may overestimate the relative risks in many settings and be misinterpreted. (NEW ENGLAND JOURNAL OF MEDICINE) |
| data exclusions | Data pre-processing steps such as transformations, re-coding, re-scaling, normalization, truncation, and handling of below detectable level readings and outliers should be fully described; any removal or modification of data values must be fully acknowledged and justified. (SCIENCE) |
| data exclusions | Each manuscript should clearly state …patients or participants with inclusion and exclusion criteria (JAMA Neurology) |
| handling missing data | Report losses to observation, such as dropouts from a clinical trial or those lost to follow-up or unavailable in an observational study. Consider multiple imputation methods to impute missing data and include an assessment of whether data were missing at random. Approaches based on “last observation carried forward” should not be used (JAMA Internal Medicine) |
| handling missing data | Clearly indicate the number of observations for each analysis or experiment, along with information about missing data. (ENVIRONMENTAL HEALTH PERSPECTIVES) |
| checking model assumptions | It is important that the author be satisfied that the assumptions behind any statistical analysis are sufficiently met and that, at least where unusual assumptions are made, unusual procedures are used, or unusual types of data are involved, and that the reader be provided with sufficient information to judge whether any departures from assumptions are severe enough to vitiate the conclusions. The amount of detail provided in any particular instance will depend on the centrality of the statistical test to the conclusions. (ECOLOGICAL MONOGRAPHS) |
| checking model assumptions | For multivariable models, report all variables included in models, and report model diagnostics and overall fit of the model when available (JAMA Neurology) |
| multiple comparisons | Reports of studies with multiple or secondary end points should address the multiple comparison issues and describe the exploratory nature of the studies. (JOURNAL OF CLINICAL ONCOLOGY) |
| multiple comparisons | Selection of endpoints. Were the primary and secondary endpoints prospectively selected? If multiple endpoints were assessed, were the appropriate statistical corrections applied?…Adjustments made to alpha levels (e.g., Bonferroni correction) or other procedure used to account for multiple testing (e.g., false discovery rate control) should be reported….more than two significant digits on p-values are usually not needed except in situations of extreme multiple testing such as in genetic association studies where stringent corrections for multiple testing might be used. (Science Translational Medicine) |
| non-parametric tests | For statistical analysis of small sample sizes, authors should use tests suitable for small sizes and provide a justification for the test used. (Cancer Discovery) |
| non-parametric tests | Normal distribution: Many statistical tests require that the data be approximately normally distributed; when using these tests, authors should explain how they tested their data for normality. If the data do not meet the assumptions of the test, then a non-parametric alternative should be used instead. (Scientific Reports) |
| null hypotheses | For null hypothesis testing, the test statistic (e.g. F, t, r) with confidence intervals, effect sizes, degrees of freedom and P value noted [LSR] (Nature Plants) |
| null hypotheses | Inherited Nature [LSR] guidance: For null hypothesis testing, the test statistic (e.g. F, t, r) with confidence intervals, effect sizes, degrees of freedom and P value noted (Scientific Data) |
| one-sided tests | State any a priori levels of significance and whether hypothesis tests were 1- or 2-sided. (JAMA Oncology) |
| one-sided tests | State whether tests were one- or two-tailed. (JOURNAL OF THE AMERICAN ACADEMY OF CHILD AND ADOLESCENT PSYCHIATRY) |
| handling outliers | If you encountered any outliers, you should describe how these were handled (eLife) |
| handling outliers | Outliers. How were outliers defined and handled? Were they defined before the beginning of the study? Have you reported outliers that were excluded?…Data pre-processing steps such as transformations, re-coding, re-scaling, normalization, truncation, and handling of below detectable level readings and outliers should be fully described; any removal or modification of data values must be fully acknowledged and justified. (Science Translational Medicine) |
| p values | Quantitative studies: abstracts should provide effect sizes with confidence intervals (not P-values alone). (BRITISH JOURNAL OF PSYCHIATRY) |
| p values | If p-values are presented, one-sided or two-sided should be specified. If one-sided, justification should be provided. Confidence intervals should also accompany the parameter for which statistical significance is being tested. (NUCLEIC ACIDS RESEARCH) |
| prespecification of analyses | All clinical trials that have begun randomization must be registered at an appropriate online public registry (see Trial Registration requirements)…Both randomized and observational studies should identify the primary outcome(s) before the study began, as well as any prespecified secondary, subgroup, and/or sensitivity analyses. Comparisons arrived at during the course of the analysis or after the study was completed should be identified as post hoc…Analyses should follow EQUATOR Reporting Guidelines and be consistent with the protocol and statistical analysis plan, or described as post hoc. (JAMA Internal Medicine) |
| prespecification of analyses | Authors are required to pre-register clinical trials with an international clinical trials register or and to cite a reference to the registration in the Methods section. Suitable databases include clinicaltrials.gov, the EU Clinical Trials Register and those listed by the World Health Organisation International Clinical Trials Registry Platform. (Nutrients) |
| sample size justification | Sample Size Calculations. For randomized trials, a statement of the power or sample size calculation is required (see the EQUATOR Network CONSORT Guidelines). For observational studies that use an established population, a power calculation is not generally required when the sample size is fixed. However, if the sample size was determined by the researchers, through any type of sampling or matching, then there should be some justification for the number sampled. In any case, describe power and sample size calculations at the beginning of the Statistical Methods section, following the general description of the study population (JAMA Neurology) |
| sample size justification | QUANTIFICATION AND STATISTICAL ANALYSIS…please summarize in this section…sample size estimation… Is there information related to experimental design? …Sample size estimation and statistical method of computation (Cell Metabolism) |
| secondary outcomes | Selection of endpoints. Were the primary and secondary endpoints prospectively selected? If multiple endpoints were assessed, were the appropriate statistical corrections applied? (Science Translational Medicine) |
| secondary outcomes | Since multiple statistical testing methods are frequently used in genotyping-phenotyping studies, please include specifics of the primary model(s) tested. Non-essential secondary models may be published as electronic data supplements. (CIRCULATION) |
| sensitivity analyses | Meta-analyses should state the major outcomes that were pooled and include odds ratios or effect sizes and, if possible, sensitivity analyses. Both randomized and observational studies should identify the primary outcome(s) before the study began, as well as any prespecified secondary, subgroup, and/or sensitivity analyses. (JAMA Internal Medicine) |
| sensitivity analyses | Meta-analyses should state the major outcomes that were pooled and include odds ratios or effect sizes and, if possible, sensitivity analyses…Both randomized and observational studies should identify the primary outcome(s) before the study began, as well as any prespecified secondary, subgroup, and/or sensitivity analyses. (JAMA-JOURNAL OF THE AMERICAN MEDICAL ASSOCIATION) |
| statistical significance | Define statistical terms, abbreviations, and symbols, if included. Avoid nontechnical uses of technical terms in statistics, such as correlation, normal, predictor, random, sample, significant, trend. Do not use inappropriate hedge terms such as marginal significance or trend toward significance for results that are not statistically significant…State any a priori levels of significance and whether hypothesis tests were 1- or 2-sided…Report basic numbers only but state if results are statistically significant or not significant; (JAMA-JOURNAL OF THE AMERICAN MEDICAL ASSOCIATION) |
| statistical significance | If necessary, present results of tests of significance, such as analysis of variance, in addition to tests of variability. (NEW PHYTOLOGIST) |
| subgroup analyses | Separate reporting of data by demographic variables, such as age and sex, facilitates pooling of data for subgroups across studies and should be routine, unless inappropriate. Discuss the influence or association of variables, such as sex and/or gender, on your findings, where appropriate, and the limitations of the data. (LANCET ONCOLOGY) |
| subgroup analyses | 8. Meta-analysis… State whether you identified potential sources of heterogeneity prior to initiating your review and analyses, and describe whether and how you carried out subgroup or sensitivity analyses or meta-regression to explore that heterogeneity. (ANNALS OF INTERNAL MEDICINE) |
Look at external guidance
| external_guidance | n_journals | guidance_type |
|---|---|---|
| consort | 95 | reporting guideline |
| arrive | 80 | reporting guideline |
| prisma | 47 | reporting guideline |
| remark | 40 | reporting guideline |
| strobe | 39 | reporting guideline |
| stard | 36 | reporting guideline |
| equator | 25 | reporting guideline |
| icmje | 21 | reporting guideline |
| cheers | 18 | reporting guideline |
| strega | 16 | reporting guideline |
| nih principles and guidelines for reporting preclinical research | 14 | reporting guideline |
| moose | 12 | reporting guideline |
| care | 11 | reporting guideline |
| miame | 11 | reporting guideline |
| coreq | 9 | reporting guideline |
| spirit | 9 | reporting guideline |
| tripod | 9 | reporting guideline |
| brisq | 8 | reporting guideline |
| squire | 8 | reporting guideline |
| gather | 7 | reporting guideline |
| sampl | 6 | reporting guideline |
| srqr | 6 | reporting guideline |
| agree | 5 | reporting guideline |
| ispor | 5 | reporting guideline |
| mibbi | 5 | reporting guideline |
| trend | 5 | reporting guideline |
| cope | 4 | reporting guideline |
| prisma-p | 4 | reporting guideline |
| aapor | 3 | reporting guideline |
| cummings & rivara 2003 (10.1001/archpedi.157.4.321) | 3 | paper |
| orion | 3 | reporting guideline |
| spirit-ai | 3 | reporting guideline |
| apa jars | 2 | reporting guideline |
| cumming et al. 2007 (10.1083/jcb.200611141) | 2 | paper |
| grips | 2 | reporting guideline |
| nlm research reporting guidelines and initiatives (https://www.nlm.nih.gov/services/research_report_guide.html) | 2 | reporting guideline |
| olsen 2003 (10.1128/iai.71.12.6689-6692.2003) | 2 | paper |
| olsen 2014 (10.1128/iai.00811-13) | 2 | paper |
| richardson & overbaugh 2005 (10.1128/jvi.79.2.669-676.2005) | 2 | paper |
| spqr | 2 | reporting guideline |
| altman et al. 1983 (10.1136/bmj.286.6376.1489) | 1 | paper |
| apa manual | 1 | publisher |
| asa (https://www.amstat.org/asa/your-career/ethical-guidelines-for-statistical-practice.aspx) | 1 | publisher |
| biosharing information resource | 1 | reporting guideline |
| boushey et al. 2006 (10.1016/j.jada.2005.11.007) | 1 | paper |
| boushey et al. 2008 (10.1016/j.jada.2008.01.002) | 1 | paper |
| bruemmer et al. 2009 (10.1016/j.jada.2009.07.011) | 1 | paper |
| consort abstract extension | 1 | reporting guideline |
| consort extended guidelines | 1 | reporting guideline |
| consort pro extension | 1 | reporting guideline |
| entreq | 1 | reporting guideline |
| fda guidelines (https://perma.cc/w2ey-msta) | 1 | reporting guideline |
| fiar | 1 | reporting guideline |
| gleason et al. 2010 (10.1016/j.jada.2009.11.022) | 1 | paper |
| gleason et al. 2015 (10.1016/j.jand.2015.03.011) | 1 | paper |
| gnosis | 1 | reporting guideline |
| harris & raynor 2017 (10.1016/j.jand.2017.03.017) | 1 | paper |
| harris et al. 2008 (10.1016/j.jada.2008.06.426) | 1 | paper |
| harris et al. 2009 (10.1016/j.jada.2008.10.018) | 1 | paper |
| harris et al. 2012 (10.1016/j.jada.2011.09.037) | 1 | paper |
| hewitt 2012 (10.1007/s10519-011-9504-z) | 1 | paper |
| hollingshead 2008 (10.1093/jnci/djn351) | 1 | paper |
| http://www.biostathandbook.com/ | 1 | paper |
| http://www.utdallas.edu/~serfling/3332/biology_statistics_made_simple_using_excel.pdf | 1 | paper |
| https://www.ncbi.nlm.nih.gov/books/nbk153593/ | 1 | paper |
| human genome epidemiology network (hugenet) guidelines | 1 | reporting guideline |
| iucr | 1 | reporting guideline |
| kempen 2011 (10.1016/j.ajo.2010.08.047) | 1 | paper |
| miqe | 1 | reporting guideline |
| motulsky 2014 (10.1124/jpet.114.219170) | 1 | paper |
| poldrack et al. 2008 (10.1016/j.neuroimage.2007.11.048) | 1 | paper |
| qhes | 1 | reporting guideline |
| record | 1 | reporting guideline |
| sheean 2011 (10.1016/j.jada.2010.10.010) | 1 | paper |
| simon et al. 2009 (10.1093/jnci/djp335) | 1 | paper |
| strenda | 1 | reporting guideline |
| strobe-me | 1 | reporting guideline |
| strond | 1 | reporting guideline |
| sullivan et al. 2016 (10.1161/jaha.116.004142) | 1 | paper |
| table 3 in dupuy & simon 2007 (10.1093/jnci/djk018) | 1 | paper |
| tidier | 1 | reporting guideline |
| zoellner & harris 2017 (10.1016/j.jand.2017.01.018) | 1 | paper |
| zoellner et al. 2015 (10.1016/j.jand.2015.03.010) | 1 | paper |
External guidance types
| guidance_type | n |
|---|---|
| paper | 29 |
| publisher | 2 |
| reporting guideline | 52 |
Our coding for the topic “prespecification of analyses” was not consistent across coders. Specifically, some coders considered guidance/instructions related to registration of clinical trials to be covered by this topic, whilst others did not. I believe the justification of those who did not was that clinical trials registration does not necessarily involve prespecification of a statistical analysis plan. That seems a reasonable justification to me, but I’d be interested to hear thoughts from the team. If we decide that advice about clinical trials registration alone does not fall into the remit of this topic, then it is relatively straightforward to examine the verbatim text we extracted and re-code the relevant cases. However, if we decide that advice about clinical trials registration does fall under the remit of this topic, then we will need to re-visit the journal instructions to authors and search for such guidance (as this was not done consistently previously).
We have recorded whether journals refer to statistical guidance from external sources, such as reporting guidelines or academic papers. However, we have not examined those external sources, and we need to decide whether to do so. If we perform extraction and coding for all 83 external sources we have identified, this is clearly a lot of extra work. We also perhaps only really interested in the most salient statistical guidance offered by journals. On the other hand, if we do not extract information from external sources, then perhaps we are missing an important aspect of statistical guidance i.e., that journals may not provide their own guidance on certain topics because they are aptly covered by external sources. An additional complication is that in some cases, the boundary of internal vs external guidance is blurred when the ‘external’ source is an academic article that appears to have been written by members of the journal’s editorial team.
Tabular data (represented in Figures 1 and 2). There is a scroll bar at the bottom of the table to see all of the scientific fields.
| Characteristic | Overall, N = 330 | AGRICULTURAL SCIENCES, N = 15 | BIOLOGY & BIOCHEMISTRY, N = 15 | CHEMISTRY, N = 15 | CLINICAL MEDICINE, N = 15 | COMPUTER SCIENCE, N = 15 | ECONOMICS & BUSINESS, N = 15 | ENGINEERING, N = 15 | ENVIRONMENT_ECOLOGY, N = 15 | GEOSCIENCES, N = 15 | IMMUNOLOGY, N = 15 | MATERIALS SCIENCE, N = 15 | MATHEMATICS, N = 15 | MICROBIOLOGY, N = 15 | MOLECULAR BIOLOGY & GENETICS, N = 15 | Multidisciplinary, N = 15 | NEUROSCIENCE & BEHAVIOR, N = 15 | PHARMACOLOGY & TOXICOLOGY, N = 15 | PHYSICS, N = 15 | PLANT & ANIMAL SCIENCE, N = 15 | PSYCHIATRY_PSYCHOLOGY, N = 15 | SOCIAL SCIENCES, GENERAL, N = 15 | SPACE SCIENCE, N = 15 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| has_guidance | 160 (48%) | 5 (33%) | 13 (87%) | 4 (27%) | 15 (100%) | 0 (0%) | 2 (13%) | 1 (6.7%) | 8 (53%) | 1 (6.7%) | 14 (93%) | 8 (53%) | 0 (0%) | 9 (60%) | 12 (80%) | 9 (60%) | 11 (73%) | 11 (73%) | 5 (33%) | 11 (73%) | 13 (87%) | 6 (40%) | 2 (13%) |
| has_external_guidance | 137 (42%) | 4 (27%) | 12 (80%) | 2 (13%) | 15 (100%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 5 (33%) | 1 (6.7%) | 13 (87%) | 7 (47%) | 0 (0%) | 8 (53%) | 12 (80%) | 8 (53%) | 11 (73%) | 11 (73%) | 5 (33%) | 5 (33%) | 13 (87%) | 3 (20%) | 1 (6.7%) |
| has_internal_guidance | 128 (39%) | 5 (33%) | 12 (80%) | 4 (27%) | 14 (93%) | 0 (0%) | 2 (13%) | 1 (6.7%) | 7 (47%) | 1 (6.7%) | 12 (80%) | 4 (27%) | 0 (0%) | 7 (47%) | 12 (80%) | 9 (60%) | 7 (47%) | 5 (33%) | 4 (27%) | 8 (53%) | 8 (53%) | 5 (33%) | 1 (6.7%) |
| has_internal_guidance_section | 93 (28%) | 2 (13%) | 11 (73%) | 2 (13%) | 9 (60%) | 0 (0%) | 1 (6.7%) | 1 (6.7%) | 5 (33%) | 1 (6.7%) | 8 (53%) | 3 (20%) | 0 (0%) | 5 (33%) | 11 (73%) | 9 (60%) | 6 (40%) | 2 (13%) | 4 (27%) | 7 (47%) | 4 (27%) | 1 (6.7%) | 1 (6.7%) |
| has_p_value | 88 (27%) | 2 (13%) | 9 (60%) | 1 (6.7%) | 13 (87%) | 0 (0%) | 2 (13%) | 1 (6.7%) | 5 (33%) | 1 (6.7%) | 10 (67%) | 3 (20%) | 0 (0%) | 3 (20%) | 5 (33%) | 8 (53%) | 6 (40%) | 2 (13%) | 4 (27%) | 4 (27%) | 7 (47%) | 2 (13%) | 0 (0%) |
| has_significance | 52 (16%) | 2 (13%) | 4 (27%) | 1 (6.7%) | 7 (47%) | 0 (0%) | 2 (13%) | 0 (0%) | 2 (13%) | 0 (0%) | 4 (27%) | 0 (0%) | 0 (0%) | 2 (13%) | 7 (47%) | 5 (33%) | 4 (27%) | 1 (6.7%) | 0 (0%) | 2 (13%) | 6 (40%) | 3 (20%) | 0 (0%) |
| has_null_hypo | 33 (10%) | 0 (0%) | 6 (40%) | 1 (6.7%) | 0 (0%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 1 (6.7%) | 1 (6.7%) | 3 (20%) | 3 (20%) | 0 (0%) | 2 (13%) | 4 (27%) | 3 (20%) | 2 (13%) | 0 (0%) | 4 (27%) | 1 (6.7%) | 1 (6.7%) | 0 (0%) | 0 (0%) |
| has_sample_size | 72 (22%) | 3 (20%) | 10 (67%) | 3 (20%) | 7 (47%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 2 (13%) | 1 (6.7%) | 5 (33%) | 3 (20%) | 0 (0%) | 4 (27%) | 11 (73%) | 5 (33%) | 5 (33%) | 1 (6.7%) | 4 (27%) | 1 (6.7%) | 4 (27%) | 2 (13%) | 0 (0%) |
| has_conf_int | 90 (27%) | 1 (6.7%) | 12 (80%) | 2 (13%) | 13 (87%) | 0 (0%) | 1 (6.7%) | 1 (6.7%) | 5 (33%) | 1 (6.7%) | 9 (60%) | 3 (20%) | 0 (0%) | 5 (33%) | 11 (73%) | 6 (40%) | 7 (47%) | 1 (6.7%) | 4 (27%) | 2 (13%) | 5 (33%) | 1 (6.7%) | 0 (0%) |
| has_effect_size | 61 (18%) | 0 (0%) | 7 (47%) | 1 (6.7%) | 10 (67%) | 0 (0%) | 1 (6.7%) | 1 (6.7%) | 3 (20%) | 1 (6.7%) | 7 (47%) | 3 (20%) | 0 (0%) | 2 (13%) | 4 (27%) | 4 (27%) | 5 (33%) | 0 (0%) | 4 (27%) | 2 (13%) | 5 (33%) | 1 (6.7%) | 0 (0%) |
| has_multi_compare | 60 (18%) | 0 (0%) | 9 (60%) | 1 (6.7%) | 6 (40%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 2 (13%) | 1 (6.7%) | 5 (33%) | 3 (20%) | 0 (0%) | 4 (27%) | 5 (33%) | 7 (47%) | 4 (27%) | 1 (6.7%) | 4 (27%) | 5 (33%) | 2 (13%) | 0 (0%) | 0 (0%) |
| has_subgroup | 30 (9.1%) | 0 (0%) | 2 (13%) | 1 (6.7%) | 9 (60%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 6 (40%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 7 (47%) | 0 (0%) | 2 (13%) | 0 (0%) | 0 (0%) | 0 (0%) | 2 (13%) | 0 (0%) | 0 (0%) |
| has_baseline_covar | 9 (2.7%) | 0 (0%) | 0 (0%) | 0 (0%) | 5 (33%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 2 (13%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 0 (0%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 0 (0%) | 0 (0%) |
| has_non_param | 10 (3.0%) | 1 (6.7%) | 1 (6.7%) | 0 (0%) | 3 (20%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 1 (6.7%) | 1 (6.7%) | 1 (6.7%) | 1 (6.7%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) |
| has_sensitivity | 11 (3.3%) | 0 (0%) | 0 (0%) | 0 (0%) | 5 (33%) | 0 (0%) | 1 (6.7%) | 0 (0%) | 1 (6.7%) | 0 (0%) | 2 (13%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 0 (0%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 0 (0%) | 0 (0%) |
| has_model_assume | 61 (18%) | 1 (6.7%) | 9 (60%) | 2 (13%) | 4 (27%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 2 (13%) | 1 (6.7%) | 4 (27%) | 3 (20%) | 0 (0%) | 4 (27%) | 11 (73%) | 7 (47%) | 5 (33%) | 0 (0%) | 4 (27%) | 1 (6.7%) | 1 (6.7%) | 1 (6.7%) | 0 (0%) |
| has_exclusion | 64 (19%) | 1 (6.7%) | 10 (67%) | 2 (13%) | 5 (33%) | 0 (0%) | 1 (6.7%) | 1 (6.7%) | 1 (6.7%) | 1 (6.7%) | 4 (27%) | 3 (20%) | 0 (0%) | 3 (20%) | 10 (67%) | 6 (40%) | 4 (27%) | 2 (13%) | 4 (27%) | 1 (6.7%) | 4 (27%) | 1 (6.7%) | 0 (0%) |
| has_outliers | 7 (2.1%) | 0 (0%) | 2 (13%) | 0 (0%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 3 (20%) | 0 (0%) | 1 (6.7%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) |
| has_missing | 18 (5.5%) | 1 (6.7%) | 1 (6.7%) | 0 (0%) | 6 (40%) | 0 (0%) | 1 (6.7%) | 0 (0%) | 1 (6.7%) | 0 (0%) | 1 (6.7%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 3 (20%) | 1 (6.7%) | 0 (0%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 2 (13%) | 0 (0%) |
| has_one_sided | 51 (15%) | 0 (0%) | 8 (53%) | 1 (6.7%) | 6 (40%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 1 (6.7%) | 1 (6.7%) | 3 (20%) | 3 (20%) | 0 (0%) | 2 (13%) | 4 (27%) | 7 (47%) | 4 (27%) | 0 (0%) | 4 (27%) | 1 (6.7%) | 4 (27%) | 1 (6.7%) | 0 (0%) |
| has_bayes | 39 (12%) | 0 (0%) | 7 (47%) | 1 (6.7%) | 2 (13%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 1 (6.7%) | 1 (6.7%) | 3 (20%) | 3 (20%) | 0 (0%) | 2 (13%) | 4 (27%) | 6 (40%) | 3 (20%) | 0 (0%) | 4 (27%) | 1 (6.7%) | 0 (0%) | 0 (0%) | 0 (0%) |
| has_secondary | 25 (7.6%) | 2 (13%) | 1 (6.7%) | 0 (0%) | 10 (67%) | 0 (0%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 0 (0%) | 4 (27%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 3 (20%) | 1 (6.7%) | 0 (0%) | 0 (0%) | 2 (13%) | 0 (0%) | 0 (0%) |
| has_prespecify | 81 (25%) | 2 (13%) | 8 (53%) | 1 (6.7%) | 14 (93%) | 0 (0%) | 1 (6.7%) | 1 (6.7%) | 1 (6.7%) | 1 (6.7%) | 7 (47%) | 3 (20%) | 0 (0%) | 5 (33%) | 6 (40%) | 8 (53%) | 5 (33%) | 2 (13%) | 4 (27%) | 3 (20%) | 8 (53%) | 1 (6.7%) | 0 (0%) |
| has_cat_continuous | 2 (0.6%) | 0 (0%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 0 (0%) |
| has_publisher_guidance | 46 (14%) | 0 (0%) | 8 (53%) | 2 (13%) | 0 (0%) | 0 (0%) | 0 (0%) | 1 (6.7%) | 2 (13%) | 1 (6.7%) | 4 (27%) | 3 (20%) | 0 (0%) | 4 (27%) | 10 (67%) | 3 (20%) | 3 (20%) | 0 (0%) | 4 (27%) | 1 (6.7%) | 0 (0%) | 0 (0%) | 0 (0%) |